Research in Higher Education Journal 


College students’ evaluation of effective teaching: Developing an 
instrument and assessing its psychometric properties 

Hamzeh Dodeen 
United Arab Emirates University 


ABSTRACT 

Students’ evaluations of teaching (SETs) are currently the most commonly used method 
for evaluating teaching effectiveness in higher education institutions. They aid in evaluating the 
quality of faculty teaching and provide useful information for administrators, faculty, and 
students. The majority of SET instruments were developed based on faculty and/or 
administrators’ knowledge and experience but excludes students’ input. The goals of this study 
were to develop a SET instrument using student samples from the United Arab Emirates 
University (UAEU), identify its dimensions, and assess its psychometric properties. Data 
collected from a total sample of 2367 students who were selected randomly and distributed in 
three different sub-samples. In the process of developing the SET instrument, two types of 
validity: content and construct, and two types of reliability: stability of the instrument over time 
and internal consistency were evaluated. A confirmatory factor analysis (CFA) was also used to 
cross-validate the developed instrument. The final version of the SET instrument consists of 29 
items in five dimensions. These five dimensions are: Teachers’ Knowledge and Organization (7 
items), Clear Explanation (6 items), Grading and Evaluation (6 items), Teaching Methods (4 
items), and Relationship with Students (6 items). The developed SET is neither short nor long 
and is not subject matter specific, so it can be used in different classrooms. 
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INTRODUCTION 

Students’ evaluations of teaching (SETs) are currently the most commonly used method 
for evaluating teaching effectiveness in higher education institutions. In fact, they receive more 
attention than do other measures of teaching such as peer evaluation, faculty self-reports or 
individual teaching portfolio (Comm & Manthaisel, 1998; Seldin, 1993). Moreover, SETs are 
often used by many institutions as the sole measure of teacher effectiveness (Washburn & 
Thornton, 1996; Wilson, 1998). They aid in evaluating the quality of faculty teaching and 
provide useful information for administrators, faculty, and students. The results obtained from 
SETs can be used for several administrative decisions such as tenure, promotion, merit pay, 
salary increase, teaching award, and contract renewal. This is the summative function or 
assessment of SETs. For faculty members, teaching evaluations provide useful feedback to 
improve their future teaching performance and facilitate growth (Rifkin, 1995). This is known as 
the formative function or assessment of SETs. Data generated by SETs can also be used to 
develop and improve courses. Finally, for students, results and information from evaluating 
faculty members assist in selecting future courses and instructors (Gary & Bergmann, 2003; 
Marsh & Roche, 1993). The use of SETs also gives students an important opportunity to 
effectively contribute to the teaching-learning process. According to a major literature review: 
“SETs are (1) multidimensional; (2) reliable and stable; (3) primary a function of the instructor 
who teaches the course rather than the course that is taught; and (4) relatively valid against a 
variety of indicators of effective teaching” (Marsh, 2001, p. 184). 

SET instruments typically contain one or more Likert-type rating scales and some open- 
ended questions that allow students to write their comments or suggestions. However, the 
essential part that is usually used in making decisions is the rating scales. Reviewing many SET 
instruments indicates that these evaluation forms cover many characteristics of good or effective 
teaching, such as teaching methods, knowledge, organization, interaction with students, clarity, 
effective communication, grading, using technology, flexibility, enthusiasm, and valuable 
feedback (Marsh & Bailey, 1993). The characteristics of effective teaching also includes fairness 
of grades, rapport with students, personal characteristics, and preparation (Jackson, et.al., 1999), 
availability of instructor outside the classroom, workload, interesting presentations, clarifying 
difficult points (Gursoy & Umbreit, 2005), and others. However, there are some characteristics 
of effective teaching that are usually not included in SET instruments. These are mostly 
characteristics that cannot be observed by students in the classroom, such as course design, 
quality of readings, academic standards, and quality of assignments (Murray, 2005). In general, 
“The validity and usefulness of SET information depends upon the content and coverage of the 
items and the SET factors that they reflect” (Marsh, 2007, p. 321). 

Research has shown that teaching effectiveness is multidimensional (Marsh, 2001; Gage 
& Berliner, 1992; Huitt, 1995). Some indicators/components (e.g., communication skills, attitude 
toward the students, knowledge of the subject, organizational skills, enthusiasm, fairness, 
flexibility, and encouragement of students) are identified to be strongly related to teaching 
effectiveness (Kim, Damewood, & Hodge, 2000). Toland & Alyala (2005) identified three 
dimensions of teaching effectiveness namely, instructor delivery of course information, 
instructor-student interaction, and regulation of students’ learning. Marsh & Roche (1997) 
identified nine dimensions: learning/value, enthusiasm, organizing, group interaction, 
relationship with students, extent of coverage, examination and grading, assignments, and 
workload/difficulty. Jackson et al. (1999) identified the following six factors: relationship with 
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students, course value, organization, grading, difficulty, and workload. Similarly, Gursoy & 
Umbreit (2005) identified four factors: learning, instruction, organization, and workload as 
effective teaching measures. Marks (2000) identified five dimensions: organization, workload, 
expected grades, teacher’s concern, and learning. In the UAE, Badri, Abdulla, Kamali, & 

Dodeen (2006) identified five factors of teaching effectiveness: knowledge and performance in 
teaching, grading, overview of the course, requirements/efforts, and course outcomes. 

Regardless of the number and construct of dimensions, it is clear that SETs assess 
distinguished components of effective teaching. However, there is no agreement on the nature 
and number of dimensions underlying students’ perceptions of teaching effectiveness (Gursoy & 
Umbreit, 2005). Therefore, understanding the multidimensionality of effective teaching is 
essential when validating instruments and interpreting final ratings. Particularly, formative 
diagnostic feedback from SETs, which intends to improve teaching and learning, should reflect 
this multidimensionality (Marsh, 2007). 

The majority of SET instruments were developed based on faculty and administrators’ 
knowledge and experience, supplemented by review of previous research (Marsh, 2007), but 
excludes students’ input (Ory & Ryan, 2001). For example, in the last 10 years, the United Arab 
Emirates University (UAEU) used three different SET instruments to evaluate teaching 
effectiveness of its faculty members. The three forms were developed by specialized ad hoc 
committees of administrators and faculty members based on their education, experience in 
teaching at the college level, and through reviewing similar SET instruments used by other 
universities. However, none of these instruments were validated or assessed by students before 
their use. In general, students are in a unique position to identify teaching effectiveness, teachers’ 
performance, and teachers’ characteristics that enhance learning. Another noteworthy 
observation here is that most research in effective teaching has used samples of U.S. students 
only (Vulcano, 2007). Researchers (e.g., L'Hommedieu, Menges & Brinko, 1990; Murray, 
Rushton, and Paunonen, 1990), recommend that one must always determine the validity and 
utility of the evaluative instrument at the local institution at which it is used. Hence, the goals of 
this study are to develop a SET instrument using student samples from UAEU, identify its 
dimensions, and assess its psychometric properties. 

METHODOLOGY 

Participants 

Data of the study were collected from the students of UAE University. The University 
has an enrolment of approximately 13,000 students. A random sample (2367 students) 
participated in this study in three different sub-samples. A brief description of each sub-sample is 
presented in the Results section. 

Development Process 

Developing the SET instrument was conducted in the following steps: 

1. Determining the Aspects of Effective Teaching 

The main aspects of effective teaching were identified through an extensive review of literature 
on teaching effectiveness and evaluation at the college level. These aspects included essential 
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characteristics of teacher performance, teacher qualities and behaviors, and classroom 
environments. 

2. Assessing Validity 

"Validity is an indication of how well an instrument actually measures what it is intended 
to measure, and helps to ensure that there are no logical errors in drawing conclusions from the 
data” (Dodeen, 2003, p. 7). When validating a SET instrument, it is recommended to collect 
evidences to support both the content and the construct validity (Marsh, 2007). Following this, 
evidences of both types of validity were assessed as follows: 

a. Content-related validity: this is the degree to which the content of the items of the 
instrument reflects the content domain of interest. Generally, content-related validity is 
established if content experts review and agree that the instrument items are representing 
the aspects of the construct to be assessed. To assess content validity of the current 
instrument, a panel of faculty members from UAEU with similar educational background 
and experience were recruited to review the instrument. 

b. Construct-related validity: this refers to the degree to which an instrument is measuring 
an intended hypothetical construct (Gay, 1996). It also involves “how well the scoring 
structure of the instrument corresponds to the construct domain” (Onwuegbuzie et al, 2007, 
p. 118). Statistically, construct-related validity can be assessed through the use of 
Exploratory Factor Analysis (EFA) procedure. 

3. Assessing Reliability 

Reliability of an instrument means the extent to which “the results could be replicated if 
the same individuals were tested again under similar circumstances” (Crocker & Algina, 1986, 
p.105). Two types of reliability were assessed: stability overtime and internal consistency: 

a. Stability Overtime - an indicator of how much the instrument is presenting 
similar results overtime. A random sample of 242 students participated in this 
analysis. The instrument was administered twice within three weeks. The 
correlation between students' answers on the two administration times was used 
to assess the instrument stability. 

b. Internal Reliability: The internal reliability of the SET instrument was assessed 
using Cronbach’s alpha. The minimum recommended level is 0.70 (Nunnally & 
Bernstein, 1994). 

4. Cross-Validating the SET Instrument 

After identifying the nature and the number of underlying dimensions that 
structure the developed SET instrument, a Confirmatory Factor Analysis (CFA) was 
employed to cross-validate the factor structure. A total of 1081 students participated in 
this analysis. 

On evaluating the results of the CFA, several fit statistics were used. Chi-square 
tests the fit between the sample covariance matrix and the matrix implied by the model. 

A large and statistically significant chi-square value indicates poor fit (Hu & Bentler, 

1995; Oliver, Jose, & Brough, 2006). As chi-square is sensitive to the sample size, it is 
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usually recommended to utilize other fit statistics that are insensitive to sample size, such 
as the comparative fit index (CFI), the goodness-of-fit index (GFI), and the non-normed 
fit index (NNFI) (Byrne, 2001). 

RESULTS 

Prior to the statistical analyses, all variables in the collected data set were screened for 
outliers or extreme values. No outliers or extreme values were identified. The data sets were also 
screened for missing values. Most of the variables had very few missing cases which did not 
affect the results. Participants with many missing data were deleted. 

Developing the Initial Version of the SET Instrument 

Through extensive review of literature on teaching effectiveness, and students’ evaluation 
at the college level, and by reviewing related research on educational measurement, 42 different 
aspects of effective teaching were identified. These aspects consist of characteristics of teacher’s 
performance, teacher qualities and behaviors, and classroom environments. The list included 
knowledge, planning, organization, interaction with students, clarity, teaching methods, effective 
communication, grading, using technology, enthusiasm, flexibility, and others. It is necessary to 
note that these attributes are just characteristics or teaching techniques that are usually correlated 
with teaching effectiveness because effective teaching itself is a construct, and there is no 
measure that for effective teaching (Marsh, 2007). The list was reviewed several times and many 
modifications were made. By the end of this step, an initial version of the SET instrument was 
developed. 

Assessing Content-Related Validity 

A group of 15 faculty members from UAEU, with different backgrounds and long 
teaching experience at the university level, was recruited to review the content-related validity of 
the present instrument. The review process included item content, wording, clarity, 
appropriateness to university students, relationships with other items, and all related issues that 
may improve the items or the instrument as a whole. All ambiguous items were either removed 
or rewritten. All notes, comments, and suggestions produced by the reviewers were carefully 
collected, analyzed, and considered. This resulted in changing, deleting, modifying or adding a 
few items to the developed instrument. By the end of this step, a revised version of the 
instrument was prepared. This version consisted of only 38 items. The importance or value of 
each of these items were assessed from students’ perspective. A 5-point Fikert scale ranged from 
Not At All Important (1) to Extremely Important (5) was applied on these items. 

Assessing Construct-Related Validity 

Exploratory Factor Analysis (EFA) procedure was used to assess the construct-related 
validity. The goal of this analysis was to identify the number and nature of the components or 
dimensions that underline the present SET instrument. A total of 1060 students responded to the 
38 items that constitute the developed SET instrument. Table 1 (Appendix) describes the 
demographics of the participating students. The student sample represented the actual 
percentages of males and females, as well as, the eight colleges in the UAE University. The 
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sample also included students from all educational levels with different GPA records that ranged 
from very low to very high. 

To determine the appropriateness of factor analysis, the Kaiser-Meyer-Olkin (KMO) 
measure of sampling adequacy and the Bartlett test of sphericity were examined. The KMO 
measure of sampling adequacy yielded an acceptable level of .972, while the Bartlett test of 
sphericity produced a chi-square value of 26848.160 with 666 degrees of freedom, which was 
statistically significant (p c.001). Thus, the set of the items together met the requirements for 
factor analysis. “To ensure that each factor identified by EFA has only one dimension and each 
attribute loads only on one factor, attributes that have factor loadings of lower than .40, 
and attributes on more than one factor with a loading score of equal to or greater 
than .40 were eliminated” (Gursoy & Umbreit, 2005, P. 96). This procedure resulted in the 
elimination of 9 items. Through factor analysis with Varimax rotation, five factors were 
identified (see figure 1 Appendix), explaining 64.50% of the variance . The five factors were: 
Teachers’ Knowledge and Organization (7 items), Clear Explanation (6 items), Grading and 
Evaluation (6 items), Teaching Methods (4 items), and Relationship with Students (6 items). The 
identified factors were graphed by a scree plot (below). The plot determined the magnitude of the 
eigenvalues of each factor in descending order. Five factors were clearly identified before the 
plot starts to level off. 

By the end of this step, a validated form of the SET instrument was developed, which 

had 29 items in five factors. The five factors, the items of each factor, and the loadings of items 

on their corresponding factor are summarized in Table 2 (Appendix) 

Assessing Reliability 


Assessing the reliability of the present SET instrument included assessing stability 
overtime and internal consistency. A random sample of 242 students (193 females and 49 males) 
was used in assessing sstability over time. The SET instrument was conducted on the 
participating students twice within three weeks. The correlation of students' responses on the two 
applications times was calculated as a measure of stability over time. The correlation coefficient 
was .65, and it was statistically significant (p<.001). 

Cronbach’s alpha was used to assess the internal reliability of the developed 
version of the SET instrument (29 items). The minimum recommended level of internal 
reliability is .70 (Nunnally & Bernstein, 1994). The value of Cronbach’s alpha was .96, 
which indicated that the instrument is internally reliable. Internal reliability was also 
calculated for each of the five components or factors and alpha values were as follows: 

Teachers’ Knowledge and Organization a = .91, Clear Explanation a = .89, Grading and 
Evaluation a = .80, Teaching Methods a = .84, and Relationship with Students a = .67. 

The reliability analysis indicated that the SET form is stable overtime, and it is internally 
reliable. 

Confirmatory Factor Analysis (CFA): 

A confirmatory factor analysis (CFA) was conducted to determine the 
plausibility of the five factor structure for the developed SET instrument. The five 
factors and their items were: Teachers’ Knowledge and Organization (7 items: 2, 3, 4, 5, 

6, 7, and 9), Clear Explanation (6 items: 10, 11, 12, 13, 14, and 15), Grading and 
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Evaluation (6 items: 16, 17, 18, 19, 20, and 21), Teaching Methods (4 items: 23, 24, 26, 
and 27), and Relationship with Students (6 items: 31, 34, 35, 36, 37, and 38). 

Data used for the CFA was the responses of a random sample of students (n = 1081). 
Table 3 (Appendix)summarizes the demographic information of these participants. Clearly, the 
actual percentages of males and females, as well as, the 8 colleges in the University were well 
represented in this sample. The sample also included students from all educational levels with 
different GPA records that ranged from very low to very high. 

The three fit statistics used in this analysis were the comparative fit index (CFI), 
the goodness-of-fit index (GFI), and the non-normed fit index (NNFI). Index values of 
.90 or higher indicate a good fit for these three measures. Also, a residual estimate that is 
usually recommended is the root mean square error of approximation (RMSEA). A value 
of less than .10 for RMSEA is considered adequate (Bentler & Bonnett, 1980). The 
results of the fit indexes were: CFI = .914, GFI = .864, NNFI = .905, while the residual 
estimates were RMSEA = .063. As all fit indexes were above the cut-off point, (expect 
GFI which is very close) this was considered an adequate fit. 

The final SET Instrument Version 

The analysis and the development steps above resulted in producing a final 
version of the SET instrument (see Table 4 (Appendix )). This version consisted of 29 
items in five dimensions. Items were given new numbers from 1 to 29, and a 5-point 
Fikert scale was used on these items which ranged from very poor (1) to very good (5). 

DISCUSSION 

Students’ evaluations play a significant role in the evaluation of teaching effectiveness at 
the university level worldwide. The goals of this study were to develop a SET instrument using 
samples of students from UAEU, identify its dimensions, and assess its psychometric properties. 
As emphasized by Marsh (2007), SET instruments should be developed using a theory in 
teaching, or an empirical technique such as factor analysis, or a systematic method to ensure 
content-related validity. Inadequately developed instruments fail to provide a comprehensive, 
multidimension evaluation, especially for diagnostic purpose. Important components of effective 
teaching could be missing from SETs. The empirical technique was used in the development of 
the present SET instrument. 

The process of developing a SET instrument included several steps and psychometrical 
procedures. Developing the items of the scale was firstly dependent on an extensive review of 
the related literature on teaching effectiveness at the university level. Then, a group of 
experienced faculty members with background in education, measurement and evaluation, or 
educational psychology, reviewed the items of the developed instrument. Next, several validation 
procedures that included content and construct validity were used. Similarly, the instrument 
reliability was assessed through several procedures and by using several student samples’ 
responses. CFA was used to cross-validate the final version of the instrument using a new 
random sample of students. 

The final version of the developed SET instrument consists of 29 items in five 
dimensions. These five dimensions are: Teachers’ Knowledge and Organization (7 items), Clear 
Explanation (6 items), Grading and Evaluation (6 items), Teaching Methods (4 items), and 
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Relationship with Students (6 items). This result corresponds to the related literature, which 
clearly indicates that students’ evaluation of effective teaching is multidimensional (Gurosy & 
Umbreit, 2005; Jackson et al.,1999; Marks, 2000; & Marsh, 2007). Several dimensions of 
teaching evaluation have been identified by researchers. For example, Toland & Alyala (2005) 
identified three dimensions: instructor delivery of course information, instructor-student 
interaction, and regulation of students’ learning. Gurosy and Umbreit (2005) found four 
components that significantly influence students’ perceptions of effective teaching: learning, 
organization, instruction, and workload. Five components: organization, difficulty, grading, 
instructor concern, and learning were identified in Mark’s (2000) study. Jackson et al. (1999) 
listed six factors: relationship with students, course value, organization, grading, difficulty, and 
workload. By comparing the results of this study with the literature, it could be observed that the 
five dimensions of effective teaching identified in this study are similar to dimensions identified 
by other studies. In particular, organization, grading, and relationship with students seem to be 
the most common dimensions in most SET instruments. 

The final version of the developed SET instrument consists of 29 items, which is neither 
short nor long, and is easy to use. It is important to observe that the developed SET instrument is 
not subject matter specific, so it can be used in different classrooms. Additional research is 
needed, however, to replicate and refine the instrument and its utility in assessing teaching 
effectiveness using different samples of university students. 
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APPENDIX 

Table 1 


Students’ demographic profde- exploratory factor analysis sample 




Number Percentage 



Number 

Percentage 

Gender 

Males 

202 

19.1 

College Humanities 

406 

38.4 


Females 

858 

80.9 

Sciences 

92 

8.7 

Year 

First 

31 

3.0 

Education 

86 

8.1 


Second 

144 

13.9 

Business 

120 

11.4 


Third 

402 

38.9 

Engineering 

101 

9.6 


Fourth 

323 

- M3- 

Faw 

128 

12.1 


Graduate 

133 

12.9 

Agriculture 

69 

6.5 

GPA 

3.50-4.00 

143 

13.8 

IT 


54 

5.5 


3.00-3.49 

259 

24.9 






2.50-2.99 

355 

34.2 






2.00-2.49 

247 

23.8 






< 2 

35 

,1-4™ 






Table 2 

Exploratory factor analysis of students’ evaluation of effective teaching 


Factor 

loadings 


Factor 1: 

Knowledge and Organization (7 items) 


Item 2 

The instructor is well-prepared in his/her course 

.608 

Item 3 

The instructor is informative when responding to students’ questions 

.622 

Item 4 

The instructor states goals and objectives clearly 

.601 

Item 5 

Fectures are well organized 

.725 

Item 6 

Effective use of class time 

.787 

Item 7 

The class time is carefully planned 

.739 

Item 9 

Effective classroom management 

.559 

Factor 2: 

Clear Explanation (6 items) 


Item 10 

Assignments, projects, activities,..etc are clear 

.442 

Item 11 

Examples are used to simplify difficult points 

.653 

Item 12 

Clear presentations of course materials 

.745 

Item 13 

Clear explanations of concepts and principles 

.750 

Item 14 

Emphasizing difficult points and facts 

.599 
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Item 15 Examination questions are clear .598 

Factor 3: Grading and Evaluation (6 items) 

Item 16 Examination cover content emphasized by the instructor .480 

Item 17 Grading criteria are clear .542 

Item 18 Offers useful feedback on assignments, projects, activities,... .684 

Item 19 Offers useful feedback on tests .712 

Item 20 His/her grading policy is fair .633 

Item 21 Uses variety of assessment methods .533 

Factor 4: Teaching Methods (4 items) 

Item 23 Uses teaching aids and technology effectively .617 

Item 24 Presents course materials at an appropriately paced sequence .537 

Item 26 Encourages students to seek knowledge from multiple resources .705 

Item 27 Motivates students to learn .627 


Factor 5: 

Item 31 
Item 34 
Item 35 
Item 36 
Item 37 
Item 38 


Interaction with Students (6 items) 

Cares for students’ learning 
Treats students with respect 

Accepts different viewpoints presented by students 
Available outside the classroom for assisting students 
Treats all students fairly 

Flexible/open-minded when dealing with students 


Table 3 

Students’ demographic profile - confirmatory factor analysis sample 


.560 

.702 

.716 

.584 

.721 

.708 




Number 

Percentage 



Number 

Percentage 

Gender 

Males 

210 

19.4 College Humanities 

417 

38.6 


Females 

871 

80.6 


Sciences 

85 

7.9 

Year 

First 

30 

2.8 


Education 

100 

9.3 


Second 

165 

15.3 

Business 

117 

10.8 


Third 

396 

36.6 

Engineering 

112 

10.4 


Fourth 

324 

30.0 

Faw 

104 

9.6 


Graduate 

146 

13.5 

Agriculture 

71 

6.6 

GPA 

3.50-4.00 

151 

14.0 

IT 

67 

6.2 


3:00-3:49 

273 

25.3 





2:50-2.99 

371 

34.3 





2:00- 2:49 

227 

21.0 





<2 

35 

3.2 





Table 4 

Students’ evaluation of teaching instrument 

Very Poor Moderate Good Very 
Poor Good 
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Knowledge and Organization 

1 The instructor is well-prepared in his/her course 

2 The instructor is informative when responding to 
students’ questions 

3 The instructor states goals and objectives clearly 

4 Lectures are well organized 

5 Effective use of class time 

6 The class time is carefully planned 

7 Effective classroom management 
Clear Explanation 

8 Assignments, projects, activities,..etc are clear 

9 Examples are used to simplify difficult points 

10 Clear presentations of course materials 

11 Clear explanations of concepts and principles 

12 Emphasizing difficult points and facts 

13 Examination questions are clear 
Grading and Evaluation 

14 Examination cover content emphasized by the 
instructor 

15 Grading criteria are clear 

16 Offers useful feedback on assignments, projects, 
activities,... 

17 Offers useful feedback on tests 

18 His/her grading policy is fair 

19 Uses variety of assessment methods 
Teaching Methods 

20 Uses teaching aids and technology effectively 

21 Presents course materials at an appropriately 
paced sequence 

22 Encourages students to seek knowledge from 
multiple resources 

23 Motivates students to learn 
Interaction with Students 

24 Cares for students’ learning 

25 Treats students with respect 

26 Accepts different viewpoints presented by 
students 

27 Available outside the classroom for assisting 
students 

28 Treats all students fairly 

29 Flexible/open-minded when dealing with student 
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